Summary

Adds an experimental backends module to ScrapeGraphAI with three alternative document loaders: Crawl4AI (async web crawler with markdown output), Obscura (CDP-based stealth browser), and Camoufox (Firefox fork with C++-level fingerprint spoofing). These backends can be selected via the new experimental key in node_config.

Also improves the core ChromiumLoader with persistent Chrome profile, storage state caching across sessions, and Cloudflare challenge detection with user guidance.

What's included

New experimental backends

• Crawl4aiLoader - uses Crawl4AI's AsyncWebCrawler for clean markdown/HTML output. Falls back to stealth Playwright + Malenia if Cloudflare blocks the crawl.
• ObscuraLoader - connects to Obscura browser or a standard Chrome instance via CDP (remote debugging protocol). Supports Docker, subprocess, and Chrome auto-start modes.
• CamoufoxLoader - starts the Camoufox server via npx, then uses its REST API for tab creation and JS evaluation.

Core improvements

• Persistent Chrome profile - ChromiumLoader now uses a persistent user data directory and caches storage state (cookies, localStorage) across sessions. This enables browser sessions to survive anti-bot challenges across restarts.
• Cloudflare detection - when Cloudflare is detected, a clear message guides users to solve the challenge once in non-headless mode (cookies are reused automatically afterward).
• Backend switching - FetchNode accepts a new experimental config key to route to the selected backend loader.

Dependency extras

pip install scrapegraphai[experimental-crawl4ai]
pip install scrapegraphai[experimental-obscura]

Camoufox does not require a Python extra (it runs via npx).

Usage

graph_config = {
    "llm": {"model": "openai/gpt-4o-mini", "api_key": "..."},
    "experimental": {
        "backend": "crawl4ai",
        "crawl4ai": {"headless": True, "output_format": "markdown"}
    }
}

Notes

• The Camoufox loader requires Node.js (npx) - see https://github.com/jo-inc/camofox-browser
• The Obscura loader requires either Docker, the Obscura binary, or a Chrome instance with remote debugging
• Existing tests pass; e2e tests require pytest -m e2e (network access)